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Who Is Doing the Engineering, the Student or the 
Teacher? The Development and Use of a Rubric to 
Categorize Level of Design for the Elementary 
Classroom 

Science, technology, engineering, and mathematics (STEM) professional 
development for K-5 teachers often includes engineering design as a focus. 
Because engineering applications provide perspective to both teachers and their 
students in terms of how mathematic and scientific principles are employed to 
solve real-world problems (Baine, 2004; Roden, 1997), there is great interest in 
using engineering as a context for studying STEM education. Engineering as a 
context for learning mathematics and science is documented in the National 
Research Council’s review of K-12 engineering curricula (National Research 
Council, 2010). Further, engineering has become integrated into the Next 
Generation Science Standards (NGSS Lead States, 2013), which provide a 
mandate for the formal integration of engineering into the K-5 curriculum. 

Although it may suggest fluidity in curriculum and instruction among the 
four disciplines, "the STEM acronym is more often used as shorthand for 
science and mathematics education” (Katehi, Pearson, & Feder, 2009, p. 12). 

The increased attention to engineering in elementary curriculum (e.g., the Next 
Generation Science Standards), teacher preparation, and professional 
development provided the motivation for our research. Our project provided 
teachers with professional development opportunities designed to enhance their 
knowledge and preparation for teaching using engineering design. Following the 
professional development course, we observed how the teachers implemented 
engineering design lessons with students in their classrooms. 

In recognition of the limited preparation of elementary level teachers to 
teach engineering content and pedagogy, we created and implemented a 
professional development opportunity for grade K-5 teachers to enhance their 
knowledge of engineering and the design process. Specifically, our collaboration 
sought to enhance the participating teachers’ understanding of the work of 
engineers. We also explored the procedures for engineering design as 
approaches to solving problems and conducting research while recognizing the 
developmentally appropriate application and use of these approaches for 
teaching STEM to elementary level learners. 
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Our STEM education intervention consisted of a three-day summer institute 
that combined presentations, workshops, hands-on activities, and curriculum 
planning and development. Our research was based on the anticipated influence 
of engineering-focused professional development on teacher practice and the 
subsequent increase in student engagement in engineering design-based learning 
activities (Fox-Tumbull, 2006). Specifically, we examined the elements of the 
design process that teachers emphasized in their instruction and the student¬ 
generated artifacts inspired by the lessons, and we classified the design 
assignments by the extent of responsibility taken by the teacher and student in 
terms of the structure of the elements in the design process. We gathered 
empirical data detailing teacher knowledge of the design process, their 
instructional use of engineering design, and student response to design 
assignments at the elementary, K-5 level. Our report presents a new level of 
design classification rubric developed for categorizing levels of responsibility of 
the students and teachers in the design process. The rubric was designed to 
classify design lessons based on the student and teacher (or instructional 
resource) responsibility for decision making in determining the structure of the 
elements of design. 

Engineering Design in Elementary Level Education 

Engineering design is becoming more popular to include as a K-5 
instructional approach, usually in conjunction with engineering, technology, and 
related contexts (Davies, 1996; Lachapelle & Cunningham, 2007). Using design 
as a focus for instruction and learning, curricular programs such as Project Lead 
the Way (Bottoms & Anthony, 2005) and Engineering is Elementary 
(Cunningham & Hester, 2007) provide structure and materials for teaching 
engineering content, concepts, and processes. Additional outreach programs and 
a growing number of teacher professional development offerings (e.g., 

Nadelson, Seifert, & Moll, 2011) have been implemented to prepare elementary 
teachers to teach using design and engineering as contexts for teaching a range 
of STEM content. Nadelson and colleagues (Nadelson et ah, 2010; Nadelson et 
ah, 2011) have established that such professional development offerings can 
increase teacher content knowledge and comfort with teaching engineering 
related content, concepts, and the design process. 

The increased interest in supporting engineering in elementary teacher 
preparation and subsequent professional development suggests that there may be 
multiple justifications for providing continuing education opportunities designed 
to enhance teacher knowledge of engineering design (Felder, Brent, & Prince, 
2011; Guzey, Tank, Wang, Roehrig, & Moore, 2014; Lewis, 2006). We 
speculate that the increase in teacher knowledge of how to engage their students 
in the engineering design process happens most effectively by engaging teachers 
in engineering design projects that adhere to student-centered instructional 
practices. 
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The majority of the research on K-12 teacher professional development in 
engineering design has been directed toward secondary teachers (Burghardt & 
Hacker, 2007; Fontenot, Talkmitt, Morse, Marcy, Chandler, & Stennett, 2009; 
Tufenkjian & Lipton, 2007). We argue that the increased awareness and 
consideration of engineering and design in the curriculum necessitates a greater 
understanding of how to prepare elementary teachers to effectively teach 
engineering design. Recognition of the potential for engineering design to 
provide learning contexts that are rich with opportunities to engage students in 
STEM habits of mind (e.g. problem solving, critical thinking, evidence based 
decision making—also see Berland, 2013) suggests that there is benefit to 
continued exploration of how design is and can be effectively taught in the K-5 
curriculum. Thus, there are a number of anticipated benefits to preparing K-5 
teachers to teach using design as well as a need to document how teachers are 
engaging their students in engineering design (Lewis, 2006). 

We recognize the need to increase teacher content and pedagogical 
knowledge associated with engineering design to enhance their capacity to 
effectively influence their students’ learning (Darling-Hammond & Bransford, 
2005). We also recognize the need to support the development of teacher 
pedagogical content knowledge (Niess, 2005; Shulman, 1987), particularly in 
the teaching of engineering (Fransson, & Holmberg, 2012), and teacher 
creativity in their STEM teaching practice. Creative expression has been deemed 
necessary for teachers to generate the mental dexterity associated with being 
flexible, adaptable, and original in their practice (Dobbins, 2009), which is 
likely critical when exploring new curricular areas such as teaching engineering 
design. Kampylis Berki, and Saariluoma (2009) argue that teachers influence 
their students’ development of creativity through modeling, which provides 
justification for enhancing and encouraging teachers to express their creativity in 
their teaching practice. Thus, a key element in providing teachers with 
experiences in design and engineering is encouraging them to think creatively 
and to reflect deeply on student-centered lessons. We assert that K-5 
engineering professional development offerings need to expose teachers to 
design as a creative endeavor. This exposure is likely to foster teachers’ creative 
expression, increasing their instructional use of design and engaging students in 
a wide range of novel, student-centered engineering challenges 

Classifying the Level of Instructional Use of Engineering Design 

One of the challenges with researching the instructional use of engineering 
design is the wide range of possible implementation configurations, from very 
teacher centered to very student centered. In order to investigate this, we needed 
to develop a tool that would allow us to classify engineering lessons with respect 
to levels of responsibility for teacher and student. A similar situation with 
scientific inquiry motivated Schwab (1962) to develop a rubric to classify the 
level of inquiry’ with respect to teacher and student responsibility. Schwab’s 
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rubric provides a means of classifying inquiry lessons based on the level to 
which the teacher or instructional resources are responsible for the structure of 
inquiry and the level to which students are responsible for the structure of the 
inquiry. In Schwab’s rubric, for a Level 0 inquiry assignment, teachers or 
instructional resources provide the structure for all elements of inquiry, and the 
students simply follow instructions as they engage in the inquiry process. Level 
0 inquiry is essentially an exercise in confirming the results from prior research 
and established outcomes. In contrast, a Level 3 inquiry (based on Schwab’s 
rubric) engages students in investigations in which they are responsible for 
generating and responding to all aspects of inquiry, a situation equivalent to full 
discovery learning. 

We were not able to locate a similar rubric or classification scheme that has 
been specifically developed to evaluate the responsibility level of students and 
teachers in engineering design instruction. We addressed this gap and developed 
and validated a rubric which can be used to classify the level of design used in 
engineering design instruction. In the development of our Level of Design 
Rubric we consulted several models of engineering design that are being 
promoted in the elementary engineering curriculum (Cunningham, 2009; 
National Aeronautics and Space Administration [NASA], 2008; The Works 
Museum, 2011). Although each of these models was unique in the 
representation of engineering design, the models share some common elements 
or processes. Common to the models were identifying problems, exploring 
ideas, brainstorming, building products, gathering data, and evaluating the 
results. Further, the models of engineering design associated with elementary 
education also recognized the iterative nature of engineering. We combined the 
common elements presented in these models with our knowledge of the 
engineering design process, adding identification and listing of constraints and 
criteria to create a framework for the critical elements of engineering design that 
should be included in K-5 lessons (see Table 1). It is important to note that we 
present the essential elements here and rely on the details of the various other 
models explained in the literature for the finer gain details (Cunningham, 2009; 
NASA, 2008; The Works Museum, 2011). 

We used the elements in our engineering design model to guide the 
development of our Level of Design Rubric. Throughout the development 
process, we made decisions to collapse some design elements from the model to 
form the principal categories of our Level of Design Rubric. For example, we 
combined the engineering design processes of generating ideas and select a 
solution into one element, and we also combined the two processes of present 
results and evaluate outcomes into a single element. The essential elements 
contained within our engineering design instructional model and a brief 
description of the processes within each of the elements are presented in Table 
1 . 
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Table 1 

Essential Elements of the Design Process Used in Instruction and the Associated 

Processes _ 

Design Element 



Problem 

Statement 

Criteria and 
Constraints 

Generate 
Ideas and 
Select 
Solution 

Process 
Used to 
Build the 
Product 

Present 
Results and 
Evaluate 


The 

Criteria to 

Brainstonning 

The 

The final 


problem to 

which the 

about possible 

solution is 

solution is 


be solved 

solution must 

solutions to 

prototyped 

presented to 


is 

conform, and 

the problem 


others 


identified 

the 


A solution 



and 

specifications 

Identifying 

is selected 

The solution 


explained 

for the 

what seems to 


is evaluated 



product are 

be the best 

A working 

for 



listed 

solution 

solution is 

conformity to 

Description 




created 

criteria and 

of the 


The 

Justification 


constraints 

Associated 


constraints, 

and assurance 

The 

and 

Process(s) 


limitations, or 

that the 

solution is 

effectiveness 



bounds for 

preferred 

tested, data 

in solving the 



the product 

solution 

are 

problem 



are 

confonns to 

gathered 




recognized 

criteria and 


Evaluation is 




constraints 


used to plan 


for the next 
generation of 
the solution 


Similar to Schwab’s (1962) rubric for classifying the level of inquiry used 
in instruction, our rubric classifies the level of engineering design used in 
instruction by the level of responsibility assumed by students and the teacher for 
the structure of the engineering design elements. In the use of the Level of 
Design Rubric, each of the five design elements are considered and scored such 
that if a teacher (or the teacher-provided resources) provides all of the structure 
of the design element, the element would be scored as a 0. By contrast, if the 
student is responsible for all of the structure of the element, the element would 
be scored as a 1. If an element in our rubric is not present in a design 
assignment, such as presenting products and evaluating results'' the element 
should be scored 0 because it is assumed that the teacher (or resource developer) 
made the decision not to include the design element process and, therefore, took 
full responsibility for that element of the design activity. We anticipate that 
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students and the teacher (or resources) will share the responsibility for the 
structure of the design process elements, so fractional scores of 1 are 
encouraged. For example, an equal sharing of the responsibility for identifying 
and developing a problem statement by the teacher (or teacher-provided 
resources) and the student may result in a score of .5 for the problem statement 
element on the rubric. After each element has been scored with a value from 0 
tol, the element scores are summed, and an overall Level of Design score is 
established. Thus, the final score for a level of design used for instruction 
evaluation and research on how elementary teachers (and other teachers) are 
structuring their engineering design lessons would be a value between 0 and 5. 


Structure 
Responsibility 
Score 0 to 1 
[If teacher or 
resources 
solely 

responsible — 
Score 0] 

[If student is 
solely 

responsible — 
Score 1] 

Design Element 

Level of 
Design 
Sum of 
Element 
Scores 
(From 

0 5) 

Problem 

Statement 

Criteria and 
Constraints 

Generate 
Ideas and 
Select 
Solution 

Process 
Used to 
Build 
the 

Product 

Present 

Results 

and 

Evaluate 

Responsibility 
for Element 
Structure Score 
(From 0-1) 








Figure 1. The Level of Design Rubric. 


As with Schwab’s (1962) rubric, because level of design is somewhat 
subjective, we constructed our level of engineering design classification scheme 
to be used and interpreted on an ordinal scale based on what the observer 
focuses on during a lesson observation. The outcome from our rubric should be 
considered as a general indicator of the level of design. For example, a sum of 
scores of 2.75 may be rounded to “level 3” engineering design lesson which 
would indicate that the student assumed slightly more responsibility for the 
elements in the engineering design assignment than the teacher (or instructional 
resources). Thus, similar to those using Schwab’s rubric to classify instructional 
level of inquiry it is up to the discretion of the users of our Level of Design 
Rubric to apply their understanding of engineering education. The conditions of 
a design lesson that is under evaluation should be taken into consideration when 
interpreting the significance of the level of design which is observed. 

To illustrate what different levels of the instructional implementation of 
design rankings might represent, we provide the following examples of design 
assignments that would be ranked from Level 0 to Level 5. The examples we 
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provide detail the level of responsibility of the student, the teacher, and expected 
design assignment outcomes. 

A Level 0 design would be equated with situations in which the student 
(through direct instruction by the teacher or instructional resources) is provided 
with all design element structure for all aspects of the design activity. Thus, a 
Level 0 design assignment would have a highly prescriptive structure with 
students following provided directions to replicate the construction of an 
existing solution or structure. In a Level 0 design assignment, the teacher (or 
instructional resources) is responsible for the structure of all design elements. A 
Level 1 design assignment would be structured such that students have some 
(but rather little) responsibility for the design elements structure and thus would 
be engaging in a design process that is likely to permit them to make slight 
modifications to an expected outcome or process. In a Level 1 design, the 
deviations the students may be permitted to take would be slight, and 
expectations would be for the students to essentially follow a detailed process 
that allows minimal decision making. At Level 2, the students assume some 
responsibility for the design process elements, which may allow them to make 
significant alternations toward the development of an expected outcome or 
product. A Level 2 design assignment would allow students to take 
responsibility for the design product as long as the spirit and outcome of the 
assignment design product is maintained. At Level 3, the students are given 
more responsibility for finding a solution to a problem and are afforded the 
opportunity to add new features and alternative solutions to a predetermined 
outcome. Thus, at Level 3, students have some freedom and responsibility to 
explore possibilities and seek unique solutions within the bounds of a proposed 
problem. At Level 4, the students assume a great deal of responsibility for the 
design process and develop new ideas and designs to solve a problem they have 
identified that is likely to lack a solution or model. At Level 4, students are 
approaching the work of professional engineers, but they are provided some 
structure and direction for their work. A Level 5 design assignment would place 
the full responsibility for all of the design elements on the students with minimal 
to no structure provided by the teacher (or instructional resources). Hence, in a 
Level 5 design assignment, the teacher’s role is almost exclusively shifted to 
that of a resource guide or consultant while the students take responsibility for 
seeking and identifying problems, designing and building unique solutions to 
meet the criteria and constraints that they identified, and evaluating, 
scrutinizing, and making modifications to their products to optimize their 
designs, as a professional engineer might do. 

We do not advocate for a specific level of design but rather the right level of 
design for the corresponding level of instruction to meet student learning needs 
and developmental capacity. Similar to Schwab’s (1962) level of inquiry rubric, 
we envision our Level of Design Rubric as a way of determining the extent to 
which engineering design lessons are student centered or teacher centered. The 
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degree of responsibility for a lesson interpreted in the context of a level provides 
a means of determining the appropriate structure of an engineering design lesson 
and the possible needs for curriculum and teacher professional development to 
assure an appropriate engineering design lesson structure. Further, the Level of 
Design Rubric provides a means of documenting how engineering design 
assignments evolve over time as students become more familiar with the 
elements of design. 

Learning and the Level of Design 

Some may argue that students are likely to learn more at higher levels of 
design because of the necessity for a greater degree of engagement due to higher 
levels of responsibility. However, we maintain that the instructional success of a 
design lesson is directly associated with student learning capacity; therefore, 
success is dependent on the experience of the students and the teacher with 
design, the context of the design assignment, and the prior content knowledge 
necessary to effectively engage in finding a solution to a problem. Further, we 
assert that there is no guarantee that student engagement in an assignment and 
associated learning will be positively correlated with the level of design; 
students may become highly engaged in lower level design or become 
disengaged with higher levels. The key to student engagement and learning may 
be the alignment between the level of design and the capacity and knowledge of 
students to effectively complete an engineering design challenge. Our Level of 
Design Rubric could provide a means of documenting factors associated with 
the alignment between level of responsibility and students engagement in an 
engineering design assignment. 

We think it is wise for teachers to implement lower level design lessons 
when initially exposing their students to engineering design because students 
need to learn the processes. To help students learn the process, it is most 
effective if the teacher assumes a greater level of responsibility for the structure 
of the design and models or scaffolds the design elements for the students 
(Bruning, Schraw, Norby, & Ronning, 2004). Thus, a lower level design may be 
necessary to help student learn the steps of the engineering design process. 
Similarly, we suggest that when teachers use engineering design to introduce 
new concepts or content, they should consider using lower level teacher- 
centered engineering design lessons. Removing the responsibility from the 
students for providing the structure for the engineering design elements lowers 
the cognitive demand on the students so that they may attend to learning the new 
concepts or content without also having to attend to the design process, which 
may reduce their capacity for learning new content (Bruning et al., 2004). 

As students gain a deeper understanding of engineering design and are 
given assignments in which they can elaborate on their prior knowledge, they 
are more likely to be able to take increased responsibility for design elements. 
Thus, we maintain that engineering design as an instructional approach is likely 
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to be most effective and result in the greatest learning success when teachers 
scaffold and adjust the level of student responsibility for elements based on the 
students’ content knowledge and design experience. Our Level of Design Rubric 
could be used to document the evolution of lessons and determine how lessons 
are being structured in conjunction with other tools that document student 
learning or performance. 


Methods 

Research Questions 

The goals of our research project were to document increases in teachers’ 
knowledge of engineering design and how teachers use the engineering design 
process to teach STEM content. Of particular interest to us were the engineering 
design elements that the teachers selected, how they structured the lessons they 
developed, and how the lessons were taught in terms of levels of responsibility. 
Thus, we recognized the need to develop and use a tool to document the level of 
design of the lessons that the teachers created and taught. We formed the 
following questions to guide our research: 

• Did our participating teachers’ knowledge of engineering design 
increase due to participation in our summer institute, and if so, was the 
shift sustained over time? 

• What was the challenge focus of the engineering design lessons that the 
teachers taught? 

• What elements of engineering design did the teachers emphasize in 
their lessons? 

• What was the level of engineering design of the observed lessons? 

• How did the students engage in the engineering design lessons? 

We speculated that our participating teachers would experience a sustained 
increase in their knowledge of engineering design due to their participation in 
our summer professional development institute and the follow-up support for 
teaching engineering design that they received during the school year. Further, 
we anticipated that our participating teachers would develop a diversity of 
creative engineering design lessons that utilized all the basic design elements. 
We predicted that we would find that the teachers had implemented engineering 
design lessons in a range of levels of design. Based on the structure and context 
of engineering design lessons, we speculated that the students would be highly 
engaged in the design lessons. 

Participants. Our data were drawn from observations of the 142 K-5 
elementary teachers who voluntarily participated in our STEM-focused 
professional development project. The participants all worked in the same 
school district in one of six partnering elementary schools. The mean age for 
teachers was 40.7 years old (S = 10.2) with 10.5 years of teaching experience (S 
= 7.4). Ninety percent were female. The participants had completed an average 
of 3.6 college level mathematics classes and 3.2 college level science classes. 
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Eighty-four percent declared a major endorsement in elementary education with 
other relevant major endorsements including biology or life science (3%), 
physical science (1%), and mathematics (1%). The participants’ teaching 
assignments were nearly equally distributed among Kindergarten to Grade 5. 

During the three years of our professional development program, the 
participants voluntarily engaged in a free, three-day intensive summer institute 
for which they received continuing education credit. For the small minority of 
cases in which teachers did not attend the professional development program, 
we provided an equivalent professional development opportunity for them at 
their schools. In addition, the participants received mentoring and follow-up 
professional development during the academic year. Because our summer 
institute was not focused specifically on engineering and design but rather on 
STEM in general, not all of our summer institute participants are included in this 
report of our research project. Thus, our current report of design lesson structure 
and content is based on observation data collected on our project participants 
who were teaching engineering design lessons on the days we conducted 
classroom observations. 

STEM Professional Development Summer Institute 

Our three-day professional development summer institute was designed to 
enhance the participating K-5 teachers’ knowledge of STEM content as well as 
their use of scientific inquiry and engineering design as instructional 
approaches. The institute theme and content were focused on exploring the 
processes used by STEM professionals in their work as contexts for teaching 
and learning in K-5 education. A portion of the institute was dedicated to 
exploring the use of plastic brick manipulatives (Lego 8 -like bricks marketed as 
PCS BrickLah' by PCS Edventures!) for teaching STEM. Our primary goal was 
to provide a curriculum to our participants that allowed them to gain a deeper 
understanding of the best practices used to teach STEM and, in particular, to 
increase their knowledge and capacity to teach engineering design. 

During the summer institute, we engaged the teacher participants in a 
number of engineering design activities, including making the most efficient 
paper helicopter, building the tallest structure possible on an inclined plane, and 
creating a “lander” that, when released from a third floor balcony, was to slowly 
descend to a target on the ground floor (Carpinelli, Kimmel, & Rockland, 2014). 
These and other activities provided explicit instruction and models of 
engineering design and provided many opportunities for the participants to think 
about and plan for how to teach engineering design. Further, to enhance 
participants’ STEM pedagogical content knowledge, we explicitly compared 
and contrasted the goals and processes of inquiry and design and encouraged the 
teachers to think about creative and unique solutions and possible opportunities 
for implementation of engineering design within their curriculum. 
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Following the summer institute, the participants were expected to develop 
and implement four lessons that utilized the plastic brick manipulatives to teach 
lessons from each of the four STEM domains. Teachers were encouraged to get 
creative, to adapt and adopt extant lessons, and to develop interesting and 
engaging STEM learning opportunities for their students. We observed the 
participants at least twice during the academic year while they were teaching a 
STEM lesson (approximately 30-60 minutes in length) to their students using 
the plastic brick manipulatives. We provided the teachers with observation data 
and feedback, extending beyond the summer institute, in order to situate the 
professional development in the classroom. 

Data Collection 

Design Process Knowledge. To assess our participants’ knowledge of 
engineering design, we adapted and adopted items from an extant instrument, 
the Design Process Knowledge Test, which had been validated for 
undergraduate engineering majors (Sims-Knight, Upchurch, & Fortier, 2006). 
The original instrument used a selected response format to assess engineering 
design knowledge across a range of related concepts. Because of the difference 
in our study population (K-5 teachers) as compared with the original instrument 
targeted population (undergraduate engineering majors), we determined it 
necessary to screen the items in the instrument and select only those that were 
aligned with general knowledge of engineering design and remove items 
associated with idiosyncratic engineering definitions, engineering coursework, 
or engineering degree program structures or activities. The resulting instrument 
contained 18 items such as, “Which of these is the best definition of engineering 
design?” and “Which is not a benefit of preliminary design or prototype?” and 
alternatives that represented a range of possible views from naive or 
misconceived to informed. Our version of the instrument also included several 
items such as, “Successful design involves breaking a problem down into 
smaller problems” which required responses along a Likert-type scale ranging 
from 1 (almost always true) to 5 (almost always false), including 1 don’t know’. In 
the original study, Sims-Knight, Upchurch, and Fortier (2006) report a 
Cronbach’s alpha of .84, indicating a good level of instrument reliability. We 
gave participants our modified version of the knowledge of design instrument as 
a pre and posttest. 

Classroom Observations. Following the summer institute, we conducted 
classroom observations of the participants to determine the extent to which they 
utilized the summer institute concepts to teach STEM content. We used an 
observation rubric that we successfully used in prior research (Nadelson et ah, 
2010) to document our observations. The observation rubric was structured to 
assure that data collection was consistent in that the same kinds of data were 
collected during each observation. The observation rubric was also structured to 
be flexible enough to document variations in the nature of the learning and 
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instruction that took place that we could associate with our summer institute 
content. Each lesson was observed in its entirety, most of which were about 50 
minutes in duration. It was our goal to observe each participant teach two STEM 
lessons, but it was up to the teacher to select what area of STEM lesson was 
taught and the STEM pedagogy that was used in the lesson. Thus, the STEM 
content area under observation was not restricted to an engineering design 
lesson but could be from any of the STEM domains. 

During the observations, our field researchers did not participate in the 
instruction, but they may have interacted with the students by asking the 
children to explain what they were doing, which allowed for the accurate 
completion of the observation rubric. We also utilized Livescribe Smartpens 1 ' to 
audio record the observed lessons to provide an additional means of accurately 
documenting the lessons for the completion of the observation rubric. The audio 
recordings were not retained once the observation rubrics were completed. 

In some circumstances, video recordings were made of the students’ 
interactions while building and explaining their products. We were authorized to 
gather the video as long as student names, faces, or other information was not 
gathered that would allow the students to be readily identifiable. These video 
recordings were also useful for completing the observation rubrics. 

Classification of the Level of Design. We utilized our Level of Design 
Rubric to document the engineering design elements that the teachers 
emphasized in their lessons and to quantify the level of engineering design. 

Thus, we used our Level of Design Rubric to document the level of 
responsibility that the teachers (or teacher-provided instructional resources) and 
students took for the elements associated with an observed engineering design 
lesson. Again, the Level of Design Rubric score is interpreted as a basis for 
documenting the source of the responsibility for element structure in a design 
activity. If an overall level of design score is near 0, indicating a low level of 
design, it is because most (or all) of the responsibility for the design structure 
was provided by the teacher (or instructional resources). In contrast, if a level of 
design score is near 5, indicating a high level of design, a major amount (or all) 
of the responsibility for the structure for the design activity was assumed by the 
student. 

To establish the inter-rater reliability of the instrument we had two 
researchers independently score the lessons. The level of agreement was 
approximately 85%, with differences resolved though conversation. Thus, we 
were confidence that the 85% level of inter-rater reliability was acceptable and 
that our data could be examined without concerns of consistency or bias. 

Results 

Teacher Knowledge of Design 

Our first research question asked: Did our participating teachers ’ 
knowledge of design increase due to participation in our summer institute and if 
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so was the shift sustained over time? Before we conducted our analysis, we 
calculated the reliability of our modified design process instrument which we 
found to have a Cronbach’s Alpha of .78, indicating that we had an acceptable 
level of instrument reliability and could proceed with our analysis with the 
assumption of consistent measures of our participants’ knowledge of design. To 
answer our research question, we conducted a paired samples r-test using our 
participants’ pretest, immediate posttest, and delayed posttest scores. Our paired 
samples t-test analysis revealed a significant increase in design knowledge t (46) 
= 4.94 ,p < .01, with the pre-institute composite scores 10.17 (5= 3.60) shifting 
upward post-institute to 12.91 (5 = 2.07), a .35 partial eta squared effect size. 
Our analysis did not reveal a significant change in design knowledge from 
posttest to delayed posttest. Our results indicate that our participants 
experienced significant and sustained gains in their knowledge of the 
engineering design process. This finding led us to consider how the participating 
teachers transferred their knowledge of and experience with the design process 
into their instruction and interactions with their students. 

Lesson Content 

Our second research question asked: What was the challenge focus of the 
engineering lessons that the teachers taught? To answer this question, we 
conducted a content analysis of our 169 STEM lesson observations, 36 of which 
were categorized as engineering design lessons that included a design challenge. 
The balance of the lessons focused on other content areas such as mathematics 
or science. Our content analysis of the observed lessons revealed the teachers 
implemented an array of design challenges, representing a diversity of creative 
expression and focusing on a range of topics. For example, in one lesson, 
students were challenged to design a container to hold the largest number of 
Silly Bandz given certain budget constraints. Our analysis exposed a number of 
design lessons in which students were presented with challenges of building 
model structures such as houses and bridges. For example, in one lesson, the 
students were challenged to construct a bridge that could withstand a simulated 
lateral motion earthquake. In another bridge activity, the students were 
instructed to build a bridge that could span two desks. Thus, our observation 
data analysis revealed that the teachers utilized a range of topics and ideas to 
engage the students in design challenges and were creative in their selection and 
organization of their lessons. 

Our analysis also revealed that several teachers implemented lessons 
focused on developing models of existing objects or structures. For example, we 
observed several lessons in which the teacher instructed the students to “design” 
and build an element from nature, such as a flower, which we interpreted as the 
process of constructing a model of a flower. Similarly, in another lesson, 
students were instructed to design a model of the life stages of a pumpkin. The 
modeling lessons raised an unanticipated situation, provoking questions as to 
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how to classify the lessons and whether it was necessary to distinguish between 
engineering design lessons and modeling lessons in our analysis. The overlap 
between the problem-solving strategies and the stages of development 
(elements) that the teachers or students used in the model construction and 
analysis and the engineering design process provided us with justification for 
grouping the modeling lessons with the engineering design lessons. The 
grouping of the modeling and design lessons is particularly defensible when 
examining the structure of the lessons. The observed instructional steps that 
teachers implemented in their modeling lessons were essentially the same as 
those that we observed in the teachers implementing design lessons. However, 
we also explicitly recognized that the teachers in our project mixed the processes 
of model building and engineering design and creatively utilized the elements of 
design as pedagogical approaches for a variety of lesson orientations. 

Emphasized Elements 

Our third research question asked: What elements of engineering design did 
the teachers emphasize in their lessons? To answer this question, we did a 
content analysis of our classroom observations, seeking evidence of the 
elements of the design process that were emphasized in the lessons. Our analysis 
revealed that all of the lessons included the problem statement , generate ideas 
and select solution , and process used to build product elements. The 
establishing criteria and/or constraints element and the present results and 
evaluate were present in only approximately half of the observed lessons, yet, 
our analysis of the lessons by design and modeling did not reveal significant 
shifts in the presence of the elements. Thus, our analysis indicates that teachers 
consistently placed emphasis on some design elements but were less uniform in 
their implementation of other design elements. 

Level of Design 

Our fourth research question asked: What was the level of design of the 
observed lessons? To answer this question, we used our Level of Design Rubric 
to analyze the observations and determine the level of design used on the 
lessons. We analyzed all 36 of the observed design lessons and scored the 
observations (see Table 2). We used the criteria that we established in our 
introduction of the rubric, using 0 for instances when the teacher (or 
instructional resources) provided the element structure and 1 when the students 
were responsible for all the structure with values between 0 and 1 corresponding 
to mixed levels of responsibility. We then summed the outcomes of the analysis 
of the individual lessons and calculated the average to produce an overall mean 
(and standard deviation) for each of the elements. In addition, we identified the 
maximum and minimum values observed. 
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Table 2 


Average Level of Design Scoring of Our 36 Design Lessons 


Structure 
Responsibility 
Score 0 to 1 
[If teacher or 
resources 
solely 

responsible — 
Score 0] 

[If student is 
solely 

responsible — 
Score 1] 

Design Element 

Level of 
Design 
Sum of 
Element 
Scores 
(From 
0-5) 

Problem 

Statement 

Criteria and 
Constraints 

Generate 
Ideas and 
Select 
Solution 

Process 
Used to 
Build 
the 

Product 

Present 

Results 

and 

Evaluate 

Responsibility 
for Element 
Structure Score 
(From 0-1) 

.69 

.53 

.27 

.99 

.21 

2.68 

Average 
Element Score 

OSD) 

[min, max] 

.69 (.14) 
[-3, 1] 

.53 (.21) 
[0,1] 

.27 (.27) 

[0, .8] 

.99 (.08) 
[.5, 1] 

.21 (.30) 

[0, 1] 

2.68 

(.64) 

[1.5, 

3.7] 


Our level of design analysis revealed that, on average, there was a 
distributed range of level of element structure responsibility. For the problem 
statement , the teachers (or instructional resources), on average, took less 
responsibility, while the students assumed more responsibility. Our analysis 
indicated that there was a nearly equally shared responsibility of the structure of 
the constraints and criteria element. In terms of the level of the generate idea 
and select solution element, our analysis revealed that, on average, the teachers 
(or instructional resources) assumed a large part of the responsibility for 
providing the structure of the element. We found a similar outcome for the 
present results and evaluate element. For the process used to build the product, 
our analysis revealed that the students almost exclusively assumed all 
responsibility for this element in the design processes. 

Overall, our analysis revealed that, on average, the level of design (M = 
2.68, S = .64) documented in the observed lessons indicated a nearly equally 
shared responsibility by the teachers and students for establishing the structure 
of the design activities. The range of level of design structure of the lessons fell 
between a low of 1.5 and a high of 3.7. The range of design structure indicates 
that shared responsibility was present in all the observed lessons. Overall, the 
observed lessons extended beyond a comprehensively teacher (or resource) 
structured level of design. Similarly, there were no instances in which the 
structure for design was comprehensively the responsibility of the students. 
Thus, our observations revealed that, on average, the teachers implemented 
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design lessons sharing the responsibility for the structure of the engineering 
design elements with their students as the students engaged to design challenges. 

Student Engagement 

Our final research question asked: How did the students engage in the 
design lessons? To answer this question, we again did a content analysis of our 
classroom observation data and reviewed the video recordings taken during the 
implementation of the design lessons. Our analysis revealed high levels of 
student motivation and engagement. Beyond their excitement about engaging in 
the building of solutions, students were also eager to develop and refine their 
products. Many of the observations recorded eagerness by the students to 
explore possibilities and make modifications to enhance or optimize their 
products, a critical aspect of design that was not necessarily an explicit 
instructional aspect of the lessons. Our data also revealed high levels of pride 
and willingness to share and explain their products. In some situations, the 
students did wander off task and decided to engage in their own activities, which 
interestingly was more common in the lower level design lessons in which 
students were not required to provide the structure for the design elements. 

We have provided several video clips that typify the high levels of student 
engagement in design lessons at the end of our article. These clips provide 
insight into how the students explained and evaluated their products and, in 
some instances, suggested or made modifications to optimize their designs. 

Discussion and Implications 

With the increased emphasis on engineering as part of the K-5 STEM 
curriculum, as promoted in the Next Generation Science Standards (NRC, 

2013), there is a need to enhance teacher capacity to teach engineering design 
based lessons (Lewis, 2006; Wilson, 2013). Our results indicate that a rather 
brief professional development intervention with follow-up classroom support 
can significantly influence teacher knowledge of the engineering design process. 
Our results are promising because they suggest that teachers can gain lasting 
knowledge of engineering design with relatively brief but appropriately 
structured interventions. We attribute the increase in our participants’ 
knowledge to the format of our summer institute or the comparable in-school 
professional development and subsequent follow-up sessions, which placed the 
teachers in situations where they actively interacted in design challenges in the 
context of the classroom. The design activities conducted during the summer 
institute provided the teachers with both knowledge of the design process and an 
instructional model for implementing an engineering design lesson. We contend 
that the active engagement in design activities or equivalent is instrumental to 
increasing teachers’ understanding of design while enhancing their pedagogical 
knowledge of how to use design in teaching. Thus, modeling and engaging 
teachers in design activities appears to be a very effective way to increase both 
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their procedural and content knowledge of engineering design and, therefore, 
their preparation to teach engineering design lessons. 

The wide range of lessons that we observed suggests that our participating 
teachers were willing to seek out new ideas and creatively implement 
engineering design lessons. We find their willingness encouraging because it 
suggests that there is potentially a wide range of teacher initiated, highly 
engaging design lessons that will be developmental^, cognitively, and 
instructionally aligned for K-5 students. Our exposure of a possible conflation 
of modeling and engineering design suggests that if we want teachers to 
understand the distinction between modeling and design, we may need to be 
more explicit about the difference in the preparation of teachers to teach using 
design. It may also be possible that the teachers are creatively adopting and 
adapting design lessons and are not concerned with adhering to the engineering 
professionals’ conception of design but rather engaging their students in active 
learning. Teacher feedback about using design and their conceptions of design 
as compared to modeling is an excellent direction for future research. 

Our analysis of the level of emphasis that teachers placed on the design 
elements revealed a range of attention to the processes. As we reviewed the 
observations, it became apparent there was consistent emphasis placed on the 
problem statement, generate ideas and select solution, and process used to build 
the product elements. We speculate that these elements were consistently 
emphasized because they are closely aligned to project-based learning 
instruction, a pedagogical practice that most teachers are familiar with or use in 
their practice. We also exposed a reduced level of emphasis on the criteria and 
constraints and present results and evaluate elements. The lack of emphasis on 
these components may be due to the lack of experience of the teachers with 
using design in instruction and the importance of criteria, constraints, and 
evaluation to the design process. Perhaps additional reinforcement of these 
elements in the professional development program would result in a greater 
emphasis in the instruction. It may also be that these are the most nebulous and 
cognitively demanding elements of design and that teachers do not feel the 
students are developmentally prepared to address these as issues in their 
learning. Further, our field observers noted that the impact of time constraints of 
the classroom often seems to be involved in teacher determination of lesson flow 
and structure. Thus, the time intensive nature of these elements may deter 
teachers from implementing these aspects of design. We speculate that teachers 
may have chosen to focus on certain aspects of the design process that allow 
students to develop STEM habits of mind and were not necessarily concerned 
with students’ mastery of the design process as a whole. Again, why teachers 
emphasize certain elements and how the students’ abilities may influence their 
decisions for implementing design lessons is an excellent direction for future 
research. 
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As we applied our rubric to classify the level of design of the observed 
lessons, we found that, on average, the students and teachers (or instructional 
resources) equally shared responsibility for the structure of the activity. We 
speculate that the teachers structured their design assignments to engage 
students in the process while maintaining significant responsibility for the 
structure to assure that the students were successful at completing the task. Thus, 
it is likely that the teachers did not think that their students were ready to engage 
in the highest levels of design because of lack of experience with the process. It 
may also be possible that the teachers’ lack of experience in teaching using 
design constrained their ability to create design assignments that could give 
students most or all of the responsibility for the structure of the design activity. 
Regardless, we conjecture that as teachers’ experience in teaching using design 
increases, they will shift to implementing higher level design activities. It is 
important to note that our classroom observers and the teachers they observed 
did not use or even have knowledge of the design elements in the rubric; the 
data was extracted post-classroom observation. Using the rubric during 
observations may enhance our ability to expose conditions that could be useful 
in explaining why teachers choose the structure level for their students. Further, 
the data may also provide insight into what conditions might be in place that 
prompts teachers to develop higher level design assignments. 

The students’ enthusiasm for engaging in the design activities suggests that 
engineering design is an effective instructional format for motivating students to 
be actively involved in their learning. In our observations, it became apparent 
that design activities provide an excellent mechanism for creating rich learning 
context for engaging students in problem solving and thinking. Further, it 
appears that well-organized lessons are very effective in engaging students in 
the design activity and focusing their attention. Perhaps it is due to the students 
need for order, but student focus and attention to task was not correlated with 
the level of design or grade level. Thus, when using design as a context for 
instruction, it is critical that the activities be orchestrated, paced, and guided in 
an organized manner to maximize their potential to influence learning. 

In our review of learner engagement data we uncovered what appeared to be 
an innate desire of the students to refine and modify their builds to optimize 
their products. The motivation and interest of the students is very encouraging 
and should be capitalized upon when considering approaches to instruction that 
most effectively enhance student learning. We assert that students’ innate desire 
to modify their builds and optimize their products indicates that students are 
eager and capable of taking the responsibility for structuring the present results 
and evaluate design element. Ironically, our data also indicated that the teacher 
(or resources) provided the majority of the structure for the present results and 
evaluate design element. Thus, modification to design instruction to enhance 
student development of problem-solving and critical-thinking skills may be 
easily achieved by shifting a greater level of responsibility for the structure of 
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the present results and evaluate design element by allowing students to critically 
evaluate their work. The outcome of a shift to a greater level of student 
responsibility for the structure of the present results and evaluate design 
element is an excellent direction for future investigation. 

We also speculate that student engagement in modifications to their builds 
can be attributed to the plastic brick manipulatives (BrickLabs®) that they were 
using in the lesson which allow for a low “cost of change,” meaning that the 
nature of the bricks encourages students to add to, subtract from, or change the 
shape of their builds, making bricks an ideal platform for teaching design. 
Further, the bricks can be assembled into an array of configurations making 
them attractive as an instructional tool to address a wide range of curriculum. 

The presence of the bricks in the classrooms may also encourage teachers to be 
more exploratory and creative with design curriculum and instruction and 
encourage their students to consider alternatives in their builds. How student 
engagement is linked to the nature of the instructional materials such as 
BrickLabs® and how the materials influence student desires to evaluate and 
modify their builds is an excellent direction for future research. Further 
exploration to expose additional manipulatives that encourage similar levels of 
engagement in design would also be another fruitful avenue of investigation. 

Limitations 

Although we observed each of our participating teachers twice during the 
academic year, we had no specific criteria for the STEM lessons that they were 
to teach during the observations. As we pointed out, there may be a conflation of 
teacher understanding of modeling and design. Yet, our data do not necessarily 
expose the relationship between teachers’ focus on modeling as a design activity 
and their understanding and perceptions of engineering design. Additional 
observations along with interviews of teachers teaching modeling lesson may 
resolve this limitation and lead to a greater understanding of the teachers’ 
choices of lessons and understanding of design and modeling. 

Similarly, only about a quarter of our STEM lesson observations were 
engineering design lessons. Thus, the other three quarters of the participants 
may have taught design lessons from a different STEM perspective. Yet, the 36 
lessons we observed were not specifically selected because they were 
engineering design lessons but occurred in an arbitrary fashion such that we 
predict the observed lessons were likely representative of the study population. 
Additional observations and interviews to confirm our prediction would be an 
excellent direction for future research. 

As with all observation protocols, our rubric has limitations. Although we 
have refined our observation protocol over a three -year period it is still limited 
in terms of the data it is designed to collect. Thus, there may be other activities 
and nuanced variations in our participants’ design lesson implementation that we 
are not explicitly capturing. However, we did use an observation tool with 
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established consistency and inter-rater reliability to collect our observational 
data, which should have minimized possible variations in the perceptions of 
research personnel. Perhaps video recording the lessons and then reviewing 
them several times may expose subtle variations in observation data, which is an 
excellent direction for future research. The challenge will be gaining access to 
public school classrooms to gather video data for such research. 

Finally, as previously mentioned, we applied our Level of Design Rubric to 
the observation data that was gathered using a different protocol and instrument. 
Although the observers were able to substantiate our scoring and assessment of 
the observation data based on their experience, the data were not collected using 
the Level of Design Rubric. We were also using our general observation rubric 
to capture data from an array of STEM lessons because the STEM content of the 
observed lessons was not known prior to the observations. The application of 
our elements table and Level of Design Rubric in observations of design lessons 
is one of the goals of our continued research. 

Conclusions 

As engineering design grows as an instructional approach for teaching a 
range of STEM content it is critical that we develop a framework for preparing 
teachers to teach the content and the tools to examine the implementations. In 
our research project, we explored the effectiveness of a teacher professional 
development offering intended to enhance teacher capacity to teach engineering 
design. We developed and used some tools for classifying the elements of 
design and the level of the design in terms of teacher and student level of 
responsibility for the structure of the design lessons. We hope others will find 
our work useful as they plan professional development offerings for K-5 
teachers and then study teacher implementation of engineering design in the 
curriculum and student engagement in the design activities. In addition, we hope 
that those who use our Level of Design Rubric will provide us with feedback as 
we seek to refine and increase the accuracy of our tool for evaluating the 
instructional use of design. 
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